Method for an augmented reality character to maintain and exhibit awareness of an observer

ABSTRACT

Methods and systems for enabling an augmented reality character to maintain and exhibit awareness of an observer are provided. A portable device held by a user is utilized to capture an image stream of a real environment, and generate an augmented reality image stream which includes a virtual character. The augmented reality image stream is displayed on the portable device to the user. As the user maneuvers the portable device, its position and movement are continuously tracked. The virtual character is configured to demonstrate awareness of the user by, for example, adjusting its gaze so as to look in the direction of the portable device.

CLAIM OF PRIORITY

This application claims priority from U.S. Provisional PatentApplication No. 61/319,021 filed on Mar. 30, 2010 and entitled “METHODFOR AN AUGMENTED REALITY CHARACTER TO MAINTAIN AND EXHIBIT AWARENESS OFAN OBSERVER,” which is incorporated herein by reference in its entiretyfor all purposes.

BACKGROUND

1. Field of the Invention

The present invention relates to methods and systems for enabling anaugmented reality character to maintain and exhibit awareness of anobserver.

2. Description of the Related Art

A growing segment of the video game industry involves the use ofportable hand-held devices. Examples of such portable devices includededicated gaming devices, such as those manufactured by Sony ComputerEntertainment Inc. and others, as well as other types of portabledevices such as smartphones, PDA's, digital multimedia players, etc. Asthe computing power of such portable devices has increased, so has theability of the portable devices to handle increasingly complex programsand tasks.

For example, today's portable devices may include several differenttypes of hardware integrated into a single device. Many such devices mayinclude a color display, a camera, speakers, and an input mechanism suchas buttons, a touchscreen or a stylus.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present invention provide methods and systems for anaugmented reality character to maintain and exhibit awareness of anobserver. It should be appreciated that the present invention can beimplemented in numerous ways, such as a process, an apparatus, a system,a device or a method on a computer readable medium. Several inventiveembodiments of the present invention are described below.

In one embodiment, a method for causing a virtual character todemonstrate awareness of a user's position or activity is provided.According to the method, the user's position is detected by detectingthe position of a portable device. The portable device is substantiallycolocational with the user, and the position and movement of theportable device are controlled by the user. The portable device includesa camera and a display. A first image stream of a real-world scene iscaptured at the camera. A second image stream is generated based on thefirst image stream, the second image stream comprising an augmentedreality scene formed by inserting a virtual character into thereal-world scene as captured in the first image stream. The virtualcharacter is configured to perform an action indicating awareness of theposition or activity of the user. The second image stream is displayedon the display of the portable device.

In one embodiment, the insertion of the virtual character into thereal-world scene comprises placement of the virtual character on asurface identified within the real-world scene.

According to another embodiment, the detection of the position of theportable device comprises identification of anchors within the firstimage stream, the anchors being tracked during movement of the portabledevice. In various embodiments, the anchors may include augmentedreality tags or features within the real-world scene.

In one embodiment, the performance of an action indicating awarenessoccurs automatically, without requiring a command from the user.

In one embodiment, the action indicating awareness comprises adjusting aview direction of the virtual character.

In one embodiment, the adjustment of the view direction is in thedirection of the position of the portable device, and the method furtherincludes a method operation of continuously updating the adjustment ofthe view direction as the portable device moves in three-dimensionalspace.

In one embodiment, the adjustment of the view direction of the virtualcharacter includes adjustment of the head or eyes of the virtualcharacter so as to look in the direction of the portable device.

In another embodiment, the adjustment of the view direction is in thedirection of a position towards which the camera of the portable deviceis directed.

In an alternative embodiment of the invention, a method for causing avirtual character to demonstrate awareness of a user's position oractivity is provided. According to the method, a first image stream of areal-world scene is captured at a camera of a portable device. Theportable device is substantially colocational with the user, and theposition and movement of the portable device are controlled by the user.One or more markers in the first image stream are identified andtracked, and the position and orientation of the portable device istracked based on the tracking of the one or more markers. A second imagestream is generated based on the first image stream, the second imagestream depicting a virtual reality scene positioned with reference tothe markers in the first image stream. The virtual reality sceneincludes a virtual character configured to perform an action indicatingawareness of the position or activity of the user. The second imagestream is displayed on a display of the portable device.

In another embodiment of the invention, a method for providing userinteractivity with a virtual character is provided. According to themethod, user interactivity with a computer program is initiated. Userinteraction with a virtual character of the computer program isdetermined. sensor data from the user is detected during the user'sinteraction with the virtual character. The position of the user istracked using the sensor data. Further according to the method, aposition of the virtual character is detected during the user'sinteraction with the virtual character. And a relative positioningbetween the user's position and the position of the virtual character isidentified during the user's interaction with the virtual character. Acontext of the user's interaction with the virtual character isidentified. And a signaling attribute for the context of the user'sinteraction with the virtual character is also identified. The signalingattribute is applied to the virtual character to convey an awareness ofthe user by the virtual character.

In one embodiment, the computer program provides a virtual realityenvironment, the virtual reality environment including the virtualcharacter. And the user interactivity with the computer program includesinteractivity with the virtual environment.

In another embodiment, the computer program provides an augmentedreality environment, the augmented reality environment including thevirtual character. And the user interactivity with the computer programincludes interactivity with the augmented reality environment.

In one embodiment, the sensor data includes an image stream captured bya portable device controlled by the user. And the tracking of theposition of the user includes identification and tracking of markerswithin the image stream to enable determination of a position of theportable device.

In one embodiment, the context of the user's interaction with thevirtual character may include a location within a real-worldenvironment, a location within a virtual environment, or a stage withinthe computer program.

In one embodiment, the signaling attribute may include adjustment of thevirtual character's view direction, movement of the virtual character,or gesturing by the virtual character.

Other aspects of the invention will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1A illustrates a front view of an exemplary portable device, inaccordance with an embodiment of the invention.

FIG. 1B illustrates a rear view of a portable device, in accordance withan embodiment of the invention.

FIG. 2 illustrates a user using a portable device to view an augmentedreality scene, in accordance with an embodiment of the invention.

FIG. 3A illustrates an overhead view of a user interacting with avirtual character, in accordance with an embodiment of the invention.

FIG. 3B illustrates an overhead view of a user interacting with avirtual character, in accordance with an embodiment of the invention.

FIG. 4A illustrates a close-up view of a portable device, as it is beingused to view an augmented reality scene, in accordance with anembodiment of the invention.

FIG. 4B illustrates a close-up view of a portable device, as it is beingused to view an augmented reality scene, in accordance with anembodiment of the invention.

FIG. 5A illustrates an overhead view of a portable device beingmaneuvered to point in different directions, in accordance with anembodiment of the invention.

FIGS. 5B and 5C illustrate overhead views of a user interacting with avirtual character and a real person, in accordance with an embodiment ofthe invention.

FIG. 6 illustrates a close-up view of a portable device being used todisplay an augmented reality image stream, as viewed from theperspective of a user, in accordance with an embodiment of theinvention.

FIG. 7 illustrates a user utilizing a portable device to view anaugmented reality scene, in accordance with an embodiment of theinvention.

FIG. 8 illustrates two users utilizing portable devices to view anaugmented reality scene, in accordance with an embodiment of theinvention.

FIG. 9 illustrates an overhead view of a group of users utilizingportable devices to view an augmented reality scene, in accordance withan embodiment of the invention.

FIG. 10 illustrates a view of a room, in which a user is using aportable device to view an augmented reality scene, in accordance withan embodiment of the invention.

FIG. 11 illustrates a close-up view of a portable device, as viewed fromthe perspective of a user interacting with a virtual character, inaccordance with an embodiment of the invention.

FIG. 12A illustrates a graph of awareness versus time, in accordancewith an embodiment of the invention.

FIG. 12B illustrates a graph of awareness versus time, in accordancewith an embodiment of the invention.

FIG. 12C illustrates a graph of awareness versus time, in accordancewith an embodiment of the invention.

FIG. 13A illustrates a method for causing a virtual character to conveyawareness of a user, in accordance with an embodiment of the invention.

FIG. 13B illustrates a flow chart illustrating a method for a virtualcharacter to demonstrate that the virtual character knows where a humanis in his world, in accordance with an embodiment of the invention.

FIG. 14 illustrates a diagram showing components of a portable device,in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

The following embodiments describe methods and apparatus for a systemthat enables interactivity with an augmented reality character. Duringthe interactivity, the augmented reality character is capable ofmaintaining or exhibiting awareness of an observer. The observer, in oneembodiment, is the user of a portable device, who can view the augmentedreality character through a screen of the portable device. As the usermoves the portable device around in three dimensional space, theaugmented reality character maybe look back a the user. In oneembodiment, the augmented reality character looks back at the user bytracking the location of the portable device, as it is moved in thethree dimensional space. The time or times when the augmented realitycharacter looks back at the user depends on a program or contact of theinteractivity by the user with the augmented reality character. If theuser is interacting with the augmented reality character, by treatingthe augmented reality character as an avatar, the avatar may onlyexhibit awareness at certain points in time, depending on the contextand nature of the interactivity. Accordingly, various examples will nowbe described, which illustrate methods for enabling an augmented realitycharacter or object to demonstrate or exhibit awareness to or of somepoint in three dimensional space.

It will be obvious, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentinvention.

With reference to FIG. 1A, a front view of an exemplary portable device10 is shown, in accordance with an embodiment of the invention. Theportable device 10 includes a display 12 for displaying graphics. Inembodiments of the invention, the display 12 is utilized to show anaugmented reality character in real-time. In various embodiments of theinvention, the display 12 may incorporate any of various displaytechnologies, such as touch-sensitivity. The portable device 10 includesspeakers 14 for facilitating audio output. The audio output fromspeakers 14 may include any sounds relating to an augmented realitycharacter, such as sounds of the augmented reality character, other gamesounds, a background soundtrack, sounds from a remote player, or anyother type of sound.

The portable device 10 includes buttons 16 and directional pad 18, whichfunction as input mechanisms for receiving input from a user of theportable device. In embodiments of the invention, it is contemplatedthat any of various other types of input mechanisms may be included inthe portable device 10. Other examples of input mechanisms may include astylus, touch-screen, keyboard, keypad, touchpad, trackball, joystick,trigger, or any other type of input mechanism which may be useful forreceiving user input.

A front-facing camera 20 is provided for capturing images and video of auser of the portable device 10, or of other objects or scenery which arein front of the portable device 10. Additionally, a microphone 22 isincluded for capturing audio from the surrounding area, such as soundsor speech made by a user of the portable device 10.

With reference to FIG. 1B, a rear view of a portable device 10 is shown,in accordance with an embodiment of the invention. A rear-facing camera24 is provided for capturing images and video of objects and scenerywhich as located behind the portable device 10. The captured video fromthe camera 24 may be displayed on the display 12 in real-time, and maybe augmented with virtual objects so as to provide an augmented realityscene that is displayed to a user.

Additionally, the portable device 10 may include any of various sensorsfor detecting position and motion of the portable device 10. In someembodiments, the portable device 10 may include one or more of agyroscope, a magnetometer, and an accelerometer, to facilitate detectionof the position and movement of the portable device 10.

The foregoing exemplary portable device has been described in somedetail in order to illustrate one possible embodiment of a portabledevice in accordance with an embodiment of the invention. However,throughout this specification, reference will be made to portabledevices, of which the foregoing illustrated example is but one example.The scope of the present invention is not intended to be limited in anyway in any particular portable device, but may readily extend to any ofvarious kinds of portable devices. Examples of portable devices includeportable game devices, personal digital assistants (PDA's), smartphones,tablet personal computers, digital cameras, etc. For purposes of thisdisclosure a portable device will be understood as any device which iscapable of being maneuvered by a user and capable of displaying anaugmented reality character to the user.

With reference to FIG. 2, a user 30 is shown using a portable device 10to view an augmented reality scene, in accordance with an embodiment ofthe invention. As shown, the user 30 holds the portable device 10 so asto orient the rear-facing camera 24 towards a table 32. The camera 24captures an image stream of the table 32. This captured image stream isaugmented in real-time with a virtual character 34 so as to generate anaugmented reality image stream. The augmented reality image stream isdisplayed in real-time on a display of the portable device 10, which theuser 30 views.

The virtual character 34 is shown at FIG. 2 on the table 32 forillustrative purposes only. Obviously, the virtual character 34 does notactually exist on the table 32, but rather appears in the augmentedreality image stream being viewed by the user 30, and therefore appearsto the user 30 to be situated on the table 32 as shown in FIG. 2.Throughout this specification, reference will be made to virtualcharacters and virtual objects configured to perform various actions orexhibit certain characteristics. Those skilled in the art willunderstand this to mean that the virtual characters or objects willappear as such when viewed by an observer on an appropriate viewingdevice which enables the observer the see the virtual character orobject in an augmented reality image stream.

In one embodiment, the virtual character 34 is configured to demonstrateawareness of the position of the portable device 10. The virtualcharacter 34 may be configured to take any of various actions whichdemonstrate awareness of the position of the portable device 10. Forexample, the virtual character 34 may be configured to look in thedirection of the portable device 10, so that it appears that the virtualcharacter 34 is looking at the user 30. Or the virtual character 34 maybe configured to move in the direction of the portable device 10, sothat it appears that the virtual character 34 is moving towards the user30. In numerous other embodiments, the virtual character 34 may beconfigured to take any action (or inaction, if appropriate)demonstrating awareness of the position of the portable device 10.

Any of various technologies for generating an augmented reality imagestream may be applied without departing from the scope of the presentinvention. For example, in one embodiment, an augmented reality (AR) tagmay be placed on the table 32, and utilized as a fiduciary marker forgenerating the augmented reality image stream. The AR tag may be anobject or figure that is recognized when present in the captured imagestream of the real environment. The AR tag serves as a fiduciary markerwhich enables determination of a location within the real environment.Additionally, the perspective distortion of the AR tag in the capturedimage stream indicates the orientation of the AR tag.

Based on the identified location and orientation of the AR tag, anaugmented reality image stream may be generated by superimposing avirtual object or character having a location and orientation related tothe AR tag. For example, in one embodiment, the virtual character 34 canbe situated on top of (e.g. “standing” on the AR tag) or substantiallycolocational with the AR tag. In other embodiments, the virtualcharacter 34 may be situated proximate to or away from the AR tag. TheAR tag may be displayed within the augmented reality image stream, ormay be selectively removed from the augmented reality image stream.

In some embodiments, one or more AR tags may be utilized in the sameenvironment. The use of multiple AR tags provides for multiple referencepoints which are defined by the AR tags. Multiple AR tags which arespread throughout a real environment can enable more robustdeterminations of location and orientation. This can improve theaccuracy with which virtual objects are placed in the augmented realityimage stream, and/or may provide for a broader operational range asmultiple AR tags increase the effective area over which the portabledevice 10 can be maneuvered while still maintaining at least one of theAR tags within the field of view of the portable device's camera 24.

In other embodiments, any of various image analysis and recognitionmethods may be applied for purposes of generating the augmented realityimage stream based on the captured image stream. For example, naturalfeature tracking methods or simultaneous location and mapping (SLAM)methods may be applied to the captured image stream in order to generatethe augmented reality image stream. Natural feature tracking methodsgenerally entail the detection and tracking of “natural” features withina real environment (as opposed to artificially introduced fiducials)such as textures, edges, corners, etc. In other embodiments of theinvention, any one or more image analysis methods may be applied inorder to generate the augmented reality image stream based on thecaptured image stream. For example, a combination of AR tags and naturalfeature tracking or SLAM methods might be employed in order to producethe augmented reality image stream.

In accordance with embodiments of the invention, the position of theportable device 10 is constantly tracked. The tracking of the positionof the portable device 10 may be accomplished based on theaforementioned technologies which are utilized to generate the augmentedreality image stream. For example, an AR tag may serve as a referencemarker which enables determination of the position, orientation andmovement of the portable device 10 relative to the AR tag, based on thedetection of the perspective distortion of the AR tag within thecaptured image stream. Multiple AR tags might be employed to improve theaccuracy of the determination of the position and orientation of theportable device 10. Or in other embodiments, natural feature trackingmethods may be employed for purposes of tracking the position andorientation of the portable device 10.

Additionally, the movement of the portable device 10 may be trackedbased on information from motion sensitive hardware within the portabledevice 10, such as an accelerometer, magnetometer, or gyroscope. In oneembodiment, an initial position of the portable device 10 is determined,and movements of the portable device 10 in relation to the initialposition are determined based on information from an accelerometer,magnetometer, or gyroscope. In other embodiments, information frommotion sensitive hardware of the portable device 10, such as anaccelerometer, magnetometer, or gyroscope, may be used in combinationwith the aforementioned technologies, such as AR tags, or naturalfeature tracking technologies, so as to ascertain the position andmovement of the portable device 10.

As shown with continued reference to FIG. 2, the positions of objectswithin the real environment may be described with reference to acoordinate space. In the embodiment shown, a location in thethree-dimensional space is represented by Cartesian coordinates (x, y,z), where x and y are in the horizontal directions and z represents thevertical direction. By way of example only, the position of the portabledevice 10 as shown is represented by the coordinates (0, −2, 5).Furthermore, the apparent position of the virtual character 34 in thereal environment—in other words, as it appears to be located in theaugmented reality image stream—can be represented by coordinates of thereal environment as well. In this case, the virtual character 34 has thecoordinates (0, 3, 0). The location of the portable device iscontinuously tracked, and may be represented by continuously updatedcoordinates. Likewise, the location of the virtual character 34 may alsobe continuously determined and represented by its correspondingcoordinates. In other embodiments of the invention, any of variouscoordinate systems, such as cylindrical or spherical coordinate systems,may be applied without departing from the scope of the presentinvention.

With reference to FIG. 3A, an overhead view of a user 30 interactingwith a virtual character 34 is illustrated, in accordance with anembodiment of the invention. As shown, the user 30 is holding portabledevice 10, and views the virtual character 34 as part of an augmentedreality image stream which is being displayed on the portable device 10.The real environment which is being captured as an image stream by acamera on the portable device 10 includes a table 32. The virtualcharacter 34 is inserted into the captured image stream to form theaugmented image stream which is displayed on the portable device 10 tothe user 30. In the augmented image stream, the virtual character 34 isgenerated so as to be situated on the table 32.

As shown, the locations of objects are represented by a coordinatesystem of the real environment. In the example shown, a Cartesiancoordinate system is configured to have an origin at a corner of thetable. It will be understood that the origin of the coordinate systemmay have any location in space, and is merely provided in the presentinstance as merely one example. The position of the portable device 10is represented by the coordinates (−2, −4, 4), and the virtual character34 has an initial position at (6, 6, 0).

In accordance with an embodiment of the invention, the virtual character34 may be configured so as to exhibit awareness of the location of theportable device 10. For example, the virtual character 34 may beconfigured so as to look in the direction of the portable device 10. Orthe virtual character 34 may be configured so as to the move in thedirection of the portable device 10. For example, as shown at FIG. 3A,the virtual character 34 may move from an initial location atcoordinates (6, 6, 0) to a position closer to the portable device 10 atcoordinates (3, 3, 0). The virtual character 34 would thus appear to theuser 30 to be moving towards him, thereby demonstrating awareness of hislocation.

Moreover, the virtual character 34 may be configured to continuouslyfollow the movement of the portable device 10. As shown at FIG. 3B, theuser 30 has moved to a new location, and thus moved the portable device10 to coordinates (6, −5, 4). In one embodiment, the virtual character34 is configured to track this movement by, for example, continuing tolook in the direction of the portable device as it is moved. In anotherembodiment, the virtual character 34 may be configured to adjust itsmovements so as to continue to move in the direction of the portabledevice as it is moved. In so doing, the virtual character 34 exhibits anapparent awareness of the location of the portable device 10, and byextension an awareness of the location of the user 30.

In various embodiments of the invention, the specific action performedwhich indicates awareness on the part of the virtual character 34 of thelocation of the portable device 10 may be any action that isrecognizable by the user 30 as such. By way of example only, and not byway of limitation, such actions which indicate awareness might includethe following: looking in the direction of the portable device, movingin the direction of the portable device, making a sound, talking orotherwise communicating in the direction of the portable device or insuch a manner as to indicate awareness of the user, making a gesture,making a facial expression, etc. In accordance with various embodimentsof the invention, any action may be performed which indicates anawareness of the position of the portable device or the user by thevirtual character.

With reference to FIG. 4A, a close-up view of a portable device 10 isshown, as it is being used to view an augmented reality scene, inaccordance with an embodiment of the invention. The portable device 10is shown as it might be seen from the perspective of a user. In thebackground, a table 32 is shown, with an optional AR tag 40 situatedthereon. Also shown is a cup 42 situated on the table 32, forillustrative purposes. Items such as cup 42 may provide additionalreference points which may be utilized to track the position andmovement of the portable device 10, in accordance with methods describedabove. The table 32 is being captured as part of an image stream by acamera at the portable device 10. This captured image stream isaugmented with a virtual character 34, so as to form an augmentedreality image stream which is displayed on the display 12 of theportable device 10. Thus, as the user views the display 12, he is ableto see the virtual character 34 in the augmented reality image stream asif the virtual character 34 were present in the real environment.

The virtual character 34 is configured so as to demonstrate awareness ofthe position of the portable device 10. Thus, in accordance with oneembodiment as shown at FIG. 4A, the virtual character 34 is configuredto look in the direction of the portable device 10.

At FIG. 4B, the portable device 10 has been moved to a differentlocation in relation to the table 32. This change in the position of theportable device 10 is tracked. And as position of the portable device 10is tracked, so the virtual character 34 is adjusted to look in thedirection of the portable device 10, thereby demonstrating an awarenessof the position of the portable device 10.

With reference to FIG. 5A, an overhead view of a portable device 10being maneuvered to point in different directions is shown, inaccordance with an embodiment of the invention. As shown, the portabledevice 10 is directed so that a camera of the portable device 10 isdirected towards a table 32. Also shown is a cup 42 for illustrativepurposes. In one instance, the portable device 10 is maneuvered so as tobe positioned at a location 50 and have its camera directed towards alocation 51 on the table 32. As the camera of the portable device 10captures an image stream of the table 32, this image stream is augmentedwith a virtual character 34 so as to generate an augmented reality imagestream. This augmented reality image stream is displayed on a display ofthe portable device 10.

A view 54 of the augmented reality image stream is shown when theportable device 10 is positioned at location 50 and oriented towardslocation 51. The virtual character 34 is generated and configured so asto appear as if it were positioned at the location 51 in the augmentedreality image stream. Thus, the virtual character appears to be standingon the table 32 in the augmented reality image stream. Additionally, thevirtual character 34 is configured to exhibit an awareness of theposition of the portable device 10, and in this case, exhibits suchawareness by looking at the portable device 34.

However, the virtual character 34 may also be configured to exhibitawareness of the direction in which the portable device 10 is oriented.In accordance with one embodiment as shown, the virtual character 34 maybe configured so as to look in the direction that the camera of theportable device 10 is aimed. As illustrated with continued reference toFIG. 5, when the camera is positioned at location 52 and oriented so asto have its camera directed towards location 53, then the virtualcharacter 34 is configured to also appear to look in the same direction(i.e. towards location 53) in the augmented reality stream. This isillustrated by the corresponding view 55 of the augmented realitystream, in which the virtual character 34 has turned to face thedirection that the portable device is directed towards. In this manner,the virtual character 34 can be configured to demonstrate not onlyawareness of the position of the portable device, but also itsorientation, thus providing a user with an enhanced interactiveexperience with the virtual character. The virtual character may look atthe user when the user looks at the virtual character (i.e. directscamera of the portable device towards the virtual character's “position”in the real environment), and look in the direction that the user lookswhen the user looks away from the virtual character (i.e. directs thecamera of the portable device away from the virtual character's“position” in the real environment).

With reference to FIGS. 5B and 5C, overhead views of a user interactingwith a virtual character and a real person are shown, in accordance withan embodiment of the invention. As shown, at FIG. 5B, the user 30 holdsportable device 10, such that its camera is directed to an area adjacentto a third-party real person 56. A virtual character 34 is configured tobe located in this area, and thus the user 30 views the virtualcharacter 34 on a display of the portable device 10. The virtualcharacter 34 is configured to exhibit an awareness of the user's 30position or activity, and thus, in the embodiment illustrated, thevirtual character 34 looks in the direction of the portable device 10,thereby appearing to the user 30 to be looking at the user. At FIG. 5C,the user 30 directs the portable device 10 towards the real person 56.The virtual character 34 is configured to react to the activity of theuser 30, and in this instance, the virtual character 34 is configured tolook in the direction of an object towards which the portable device 10is oriented, in other words towards the real person 56.

With reference to FIG. 6, a close-up view of a portable device 10 beingused to display an augmented reality image stream is shown, as viewedfrom the perspective of a user, in accordance with an embodiment of theinvention. The portable device 10 includes a camera which is used tocapture an image stream of table 32, which is depicted in the backgroundat FIG. 6. The captured image stream is augmented with a virtualcharacter 34 so as to form an augmented reality image stream. In oneembodiment, the virtual character 34 is configured to demonstrateawareness of the location of the portable device 10 or the user. Thismay be accomplished by, for example, configuring the virtual character34 to look or move in the direction of the portable device 10.

Additionally, the augmented reality image stream may include additionalvirtual elements such as a tree 60 and vehicle 62. In embodiments of theinvention, the augmented reality image stream may include any kind ofvirtual element, the virtual elements being added to the captured imagestream so as to form the augmented reality image stream. In someembodiments, the virtual elements may be generated with reference toparticular locations or objects in the real environment, or may beconfigured to interact with objects in the real environment. In otherembodiments, virtual elements may exist independently of objects in thereal environment. In some embodiments, virtual elements may move andinteract with objects in the real environment. In other embodiments,virtual elements may be situated at fixed locations within the realenvironment. In some embodiments, virtual elements may replace objectswithin the real environment.

In various embodiments of the invention, any number of the real worldelements captured in the image stream upon which the augmented realityimage stream is based may be replaced by virtual elements, includingtotal replacement of the real world image stream with virtual elements.The augmented reality image stream may have any mix or relativeproportion of real world elements and virtual elements.

While embodiments of the invention are generally described in terms ofan augmented reality scene which is generated based on a captured imagestream of a real world scene, it is recognized that the principles ofthe present invention may also be applied to a virtual reality scene (noreal world elements visible) having a virtual reality character. Withcontinued reference to FIG. 6, in an alternative embodiment of theinvention, a virtual reality image stream is displayed on the display12. The virtual reality image stream is generated with reference to thereal world scene that is captured in accordance with the principlesdiscussed above. And the location and orientation of the portable device10 can be determined based on analysis of the real world image streamand other sensors as previously discussed. However, the video displayedto the user may consist exclusively of a virtual environment that ispositioned with reference to the real world environment. In this manner,a user views the virtual world via the portable device, yet the virtualworld is positionally tied to the real world, so that the user maymaneuver the portable device 10 about the virtual world in a naturalmanner. It will be understood by those skilled in the art that theprinciples of the present invention may be applied to embodiments inwhich a virtual reality scene is displayed to a user on the portabledevice 10, without departing from the scope of the present invention.

With reference to FIG. 7, a user 30 is shown utilizing a portable device10 to view an augmented reality scene, in accordance with an embodimentof the invention. The portable device 10 includes a rear-facing camera(not shown) which captures an image stream including the table 32. Thecaptured image stream is augmented with a virtual character 34, so as toform an augmented reality image stream that is displayed on display 12of the portable device 10. Additionally, the portable device 10 includesa front-facing camera 20, which captures an image stream of the user 10.This user image stream may be analyzed in real-time to determine variousparameters related to the user and to provide additional modes offunctionality and interactivity to the user.

For example, in one embodiment the user image stream may be analyzed todetermine a more accurate position of the user (as opposed toapproximating the position of the user based only on the position of theportable device 10). Or in another embodiment, the user image stream isanalyzed to determine facial expressions of the user. In one embodiment,the direction that the user is facing and/or the movement of the user'seyes are tracked through analysis of the user image stream, so as todetermine where the user is looking or facing. In another embodiment,the user image stream can be analyzed to determine gestures of the user,such as pointing, waving, etc. In still other embodiments of theinvention, physical attributes related to and of the user can bedetermined, such as eye color, skin type, clothing color, etc. Invarious other embodiments of the invention, any of various kinds ofexpressions, movements, positions, or other qualities of the user can bedetermined based on analysis of the user image stream, without departingfrom the scope of the present invention.

The virtual character 34 may be configured to demonstrate awareness ofthe various qualities which are determined from analysis of the userimage stream, as described above. For example, based on a moreaccurately determined location of the user 30, the virtual character 34might be configured to demonstrate awareness of the user's position by,for example, looking or moving in the direction of the user 30. Inanother embodiment, the virtual character may be configured to look inthe direction in which the user 30 is determined to be looking, based atleast in part on analysis of the user image stream.

In one embodiment, the virtual character 34 is configured to react tofacial expressions of the user. For example, the virtual character 34might provide appropriate responses to various expressions of the user30, such as demonstrating excitement when the user expresses happiness.In one embodiment, the virtual character 34 is configured to respond toexpressions of the user 30 by demonstrating similar types ofexpressions. For example, the virtual character 34 may be configured towave when the user 30 waves in the direction of the virtual character.The virtual character 34 may exhibit any of various facial expressionsand gestures so as to communicate an expression to the user 30 anddemonstrate awareness of the user's expressions and gestures.

In one embodiment, the virtual character 34 is configured to demonstratean awareness of where a user is pointing, as determined based onreal-time analysis of the user image stream captured by the front-facingcamera 20 of the portable device 10. As shown with continued referenceto FIG. 7, the user 30 is pointing in a direction 70. The virtualcharacter 34 may initially be looking at the user 30. The user imagestream captured by the front-facing camera 20 is analyzed to determinethe direction that the user is pointing. And the virtual character 34 isconfigured to then react to the user's pointing in the direction 70 bylooking in the same direction. In another embodiment, the virtualcharacter may be configured to move in the direction that the user 30 ispointing. In other embodiments of the invention, the virtual charactermay be configured to perform any type of action in response to the userpointing in a given direction, such as attacking an enemy or firing aweapon in the given direction, or other types of actions.

In another embodiment, the rear-facing camera of the portable device 10may be utilized to detect gestures of the user 30. For example, withcontinued reference to FIG. 7, in an alternative embodiment, the user 30may move their hand behind the portable device 10 so as to be capturedby the portable device's 10 rear-facing camera. Thus, the user's hand isincluded in the image stream captured by the rear-facing camera. Thisimage stream is analyzed to detect the presence of the user's hand aswell as interpret any gestures being made by the user's hand. Forexample, the user's hand may be gesturing in a particular manner, suchas pointing, waving, moving side-to-side, beckoning, grasping, etc.These gestures are detected and may be utilized to affect virtualobjects in the augmented reality image stream. The virtual character mayrespond to such gestures in a manner indicating awareness of the user'sactivity.

While the foregoing examples are described with reference to a portabledevice include a front-facing and rear-facing cameras, it will beapparent to those skilled in the art that the portable device mayinclude any number of cameras having various orientations for detectingactions and gestures of the user, as well as capturing image streams ofthe real environment.

With reference to FIG. 8, two users are shown utilizing portable devicesto view an augmented reality scene, in accordance with an embodiment ofthe invention. As shown, a user 80 holds a portable device 81, and auser 82 holds a portable device 83. The user 80 and user 82 stand onopposite sides of a table 32, with their respective portable devicesdirected towards the table. Each of the portable devices 81 and 83includes a camera which captures an image stream of the real environmentincluding the table and surrounding area which is in the field of viewof the camera. Each of these captured image streams is augmented withthe same virtual character 34, so as to form parallel augmented realityimage streams. The portable devices 81 and 83 are linked to each other,either directly (e.g. via a wireless connection) or indirectly (e.g. viaa supplemental device, such as a computer, gaming system, networkdevice, or some other computing device, etc. (not shown)) such that thesame augmented reality scene is shown on each of the portable devices 81and 83, but from different perspectives in accordance with theirrespective locations and orientations.

With continued reference to FIG. 8, a view 84 of the augmented realityscene as seen by the user 80 on his portable device 81 is illustrated.As shown, the virtual character 34 is looking at the user 80.Simultaneously, the user 82 sees a view 86 on his portable device 83. Asshown by the view 86, the virtual character 34 is looking away from theuser 82 and towards the user 80. In accordance with an embodiment of theinvention, the virtual character 34 may be configured to react to anddemonstrate awareness of both the users positions and actions. In theillustrated example, the user 82 makes a noise or speaks. This noise isdetected by a microphone of the user's 82 portable device 83. Thevirtual character 34 is configured to react to the noise by looking inthe direction of the noise, i.e. looking in the direction of theportable device 83, and thus demonstrates awareness of the position ofthe portable device 83, and by extension the position of the user 82.This is illustrated by the view 87, as seen by the user 82, in which thevirtual character 34 has now turned to face the user 82. Likewise, theview 85 seen by the user 80 illustrates that the virtual character 34has turned away from the user 80 and towards the user 82. In thismanner, the attention and awareness of the virtual character 34 can beshifted from one user to another user, both of whom are collectivelyviewing the same augmented reality scene.

While the foregoing example has been described with reference to twousers, in other embodiments any number of users can be viewing the sameaugmented reality environment, and the virtual character can beconfigured to demonstrate awareness of any of those users' positions andactions. By way of example, the virtual character may demonstrateawareness of the multiple users' positions and actions by looking,moving, speaking, or gesturing in the direction of a given user, ortaking any other type of action which demonstrates awareness of thegiven user's position and/or activity.

With reference to FIG. 9, an overhead view of a group of users utilizingportable devices to view an augmented reality scene is shown, inaccordance with an embodiment of the invention. A first user 90 holds aportable device 91; a second user 92 holds a portable device 93; a thirduser 94 holds a portable device 95; and a fourth user 96 holds aportable device 97. While the specific example shown illustrates fourusers holding portable devices, it will be appreciated by those skilledin the art that any number of users holding portable devices may beprovided for, in accordance with various embodiments of the invention.Each of the portable devices 91, 93, 95 and 97 includes a camera forcapturing an image stream of the real environment. In this case, thecameras of the portable devices 91, 93, 95, and 97 are generallyoriented towards a table 32. In this manner, the cameras of the portabledevices are all oriented towards the same real environment scene, andthus capture image streams of the same scene but from differentperspectives depending on the locations and orientations of the portabledevices 91, 93, 95, and 97. Each of the captured image streams from eachof the portable devices is augmented with the same virtual character 34,so as to form augmented reality image streams from the perspective ofeach of the portable devices 91, 93, 95, and 97. The virtual character34 will appear different to each of the users 90, 92, 94, and 96depending on the perspective from which each of their respectiveportable devices is being utilized to view the augmented reality scene.To facilitate this, the portable devices 91, 93, 95, and 97 are linkedto each other either directly or indirectly via a supplemental device soas to provide collective real-time viewing of the same augmented realityscene from different perspectives at each of the different portabledevices. In this manner, each of the users 90, 92, 94, and 96 is able tosimultaneously interact with the same augmented reality scene—and morespecifically, interact with the same virtual character 34.

It will be appreciated that the virtual character 34 may be configuredto react to and interact with the multiple users 90, 92, 94, and 96 invarious ways which demonstrate awareness of the users' positions. Forexample, the virtual character 34 may be configured to direct itsattention towards one of the users by looking or moving in the directionof the user. In one embodiment, this may occur based on a detectedmotion or sound of one of the users. The motion or sound of a user maybe captured by a microphone and camera of the user's portable device,configured to detect images and sounds from that particular user. As anexample, the virtual character 34 might be looking towards one of theusers. Another one of the users may wave his hand, say something, orprovide some other gesture so as to get the attention of the virtualcharacter 34. This gesture or sound is detected by the user's portabledevice, and the virtual character 34 may then turn towards that user inresponse to the gesture or sound.

In various other embodiments of the invention, the virtual character 34may be configured to take any kind of action, either of its own accord,or in response to some kind of input received from one or more of theusers, which demonstrates awareness of the position of the portabledevices and/or the users holding them. For example, in one embodiment,the virtual character 34 might look from one user to another user in asystematic fashion. Or in another embodiment, the virtual character 34might look at the same user until its attention is drawn away from thatuser and towards another user because of an action performed by thelatter user. In addition to looking or moving in the direction of agiven user, the virtual character 34 might also demonstrate awareness ofa user's position by talking in the direction of the user and/or theirportable device, changing a facial expression, moving its eyes, throwingan object, firing a weapon, or any other type of action so as todemonstrate an awareness of the position of the user.

The foregoing embodiments have been generically described with referenceto one virtual character. However, in other embodiments of theinvention, there may be any number of virtual characters, each of whichmay demonstrate awareness of the positions of any of various users whoare viewing the augmented reality scene via a portable device as hereindescribed.

Those skilled in the art will realize numerous possible applicationswithout departing from the scope of the present invention. By way ofexample only, and not by way of limitation, some possible applicationsare discussed herein. For example, in one embodiment, a virtualcharacter presented as part of an augmented reality scene may beconfigured to act as a performer that interacts with any of varioususers viewing the same augmented reality scene with respective portabledevices. Such interactions might involve looking in the direction of auser, gesturing towards a user, or other types of actions indicatingawareness of the user's position. In this manner, a group of users maywatch the same performance by a virtual character from differentperspectives, with enhanced interactivity provided by the virtualcharacter's apparent awareness of each of the users' positions.

In other related embodiments, there may be several virtual charactersthat are each configured to act as performers. Such a group of virtualcharacters might be configured to behave in a coordinated fashion, suchas in a musical group, dance group, acrobatic group, etc. Or such agroup of virtual characters might be configured to behave in a lesscoordinated fashion, or in such a manner that each of the virtualcharacters behaves in an independent manner. Each of the group ofvirtual characters could demonstrate awareness of one or more of thevarious users who are viewing the augmented reality scene via theirrespective portable devices.

Those skilled in the art will realize numerous variations in accordancewith the presently described embodiments without departing from thescope of the present invention. For example, in one embodiment, theforegoing principles may be applied so as to simulate an environmentwherein multiple users view one or more performers, wherein eachperformer may demonstrate awareness of the users' positions. In anotherembodiment, multiple users could collectively view a virtual lecturer,or a virtual panel discussion, or the like, wherein each of the virtualcharacters demonstrates awareness of the users' positions. In anotherembodiment, multiple users might collectively participate in a game andcollectively view a virtual game host, the virtual game host interactingwith the users in such a manner as to demonstrate awareness of theusers' positions. In other embodiments a virtual character could be ananimal, pet, animation-based character, human-based character, alien, orany other type of character that demonstrates awareness of users'positions.

With reference to FIG. 10, a view of a room 100 is shown, in which auser 30 is using a portable device 10 to view an augmented realityscene, in accordance with an embodiment of the invention. As shown, theroom 100 contains various elements and features, such as a picture 102,a door 104 and a window 106. These and other features are captured in animage stream by a camera of the portable device 10. The captured imagestream is analyzed so as to enable a determination of the position andorientation of the portable device 10, and to enable tracking of theportable device's movements as it is maneuvered by the user 30. Thedetermination of the position, orientation, and movement of the portabledevice 10 may be enhanced by the use of onboard devices included in theportable device 10, such as an accelerometer, magnetometer, and/orgyroscope.

The captured image stream of the real environment of the room 100 isaugmented in real-time with a virtual character 110. As shown, thevirtual character 110 is configured to demonstrate an awareness of theuser's 30 position by, for example, looking, facing, gesturing, ormoving in the direction of the portable device 10. Additionally, theaugmented reality scene may be configured to have other virtualcharacters, such as virtual character 112. Thus, in the example shown atFIG. 10, when the user 30 turns the portable device 10 to her right, theaugmented reality image stream displayed on the portable device 10 willinclude an image of the virtual character 112. This virtual character112 may also demonstrate awareness of the user's 30 position by, forexample, looking, facing, gesturing, or moving in the direction of theportable device 10. In this manner, the user 30 may experience andinteract with an augmented reality environment containing multiplevirtual characters that demonstrate awareness of the user's position. Asthe user moves about a given real environment, one or more of thevirtual characters in the augmented reality scene may interact with theuser in a manner demonstrating awareness of the user's position, and maytrack the movement of the user as she moves within the real environment.

With reference to FIG. 11, a close-up view of a portable device 10 isshown, as viewed from the perspective of a user interacting with avirtual character 34, in accordance with an embodiment of the invention.In the illustrated embodiment, the portable device 10 includes a camerawhich captures an image stream including the table 32, shown in thebackground. The captured image stream is augmented with various elementsso as to produce an augmented reality image stream, which is displayedon the display 12 of the portable device 10. As shown, the augmentedreality image stream includes a virtual track 116, and the virtualcharacter 34 is configured to drive a virtual vehicle 114 on the virtualtrack 116. The virtual character 34 is configured to demonstrateawareness of the user's position. Thus, in one embodiment, when thevirtual character 34 drives the virtual vehicle 114 towards the user orcloser to the user along the virtual track 116, the virtual character 34may look in the direction of the user. Whereas, when the virtualcharacter 34 drives the virtual vehicle 114 away from the user or isfarther from the user along the virtual track 116, then the virtualcharacter 34 may not look in the direction of the user. In various otherembodiments, the virtual character 34 may demonstrate awareness of theuser's position by performing other actions, such as driving at adifferent speed (e.g. slowing down), making a facial expression, etc.

As noted above, in an alternative embodiment, a virtual realityenvironment (as opposed to an augmented reality environment) can begenerated based on the captured image stream of the real environment. Insuch an embodiment, the virtual elements of the virtual reality imagestream are positioned with reference to elements of the realenvironment, the elements of the real environment being detected basedon analysis of the captured image stream of the real environment. Forexample, as shown of FIG. 11, the virtual track 116, virtual vehicle114, and virtual character 34 are positioned with reference to elementsof the real environment, such as the table 32. A virtual reality imagestream which depicts the virtual reality environment is generated anddisplayed on the display 12 of the portable device 10. The user thusviews a virtual reality scene rather than the real-world scene on theportable device 10, but is able to move the portable device 10 todifferent positions within the virtual environment by physicallymaneuvering the portable device 10, as the virtual environment ispositioned with reference to the real environment which is beingcontinuously captured by the portable device's camera.

In various embodiments of the invention, there may be different levelsof awareness of a user's position demonstrated by a virtual character.For example, in some embodiments, the quality of a particular action ofthe virtual character which demonstrates awareness may be variable. Forexample, the percentage of time that a virtual character looks at a userin such a manner as to demonstrate awareness of the user's position mayvary. Or the extent to which a virtual character moves in a user'sdirection may vary. In other embodiments, any quality associated with anaction demonstrating awareness of a user's position may be variable soas to provide different levels of awareness exhibited by the virtualcharacter.

Additionally, different levels of awareness may be defined by variouscombinations of activities which demonstrate awareness of a user'sposition. For example, one level of awareness may be defined by thevirtual character looking in the direction of the user. Whereas, anotherlevel of awareness may be defined by a combination of the virtualcharacter looking in the direction of the user as well as moving in thedirection of the user. It will be appreciated by those skilled in theart that in various embodiments of the invention, any of variouscombinations of actions indicating awareness of a user's position may beused to define different levels of awareness. Furthermore, both aquality of an action demonstrating awareness, and combinations of suchactions may be integrated in order to define various levels of awarenessof a user's position which may be exhibited by a virtual character.

While the various levels of awareness have been described with referenceto the performance of particular actions, it will be appreciated thatlevels of awareness may also indicate variations in the ability to takea particular action which may indicate awareness of the user's position,regardless of whether the particular action is actually performed by thevirtual character or not. Thus, for example, one level of awarenessmight be defined by the virtual character having the ability to look inthe direction of the user. However, the performance of the virtualcharacter actually looking in the direction of the user might not occurunless triggered by an activity of the user, such as a movement,gesture, sound, or other activity. In various embodiments of theinvention, a level of awareness can be defined for a virtual characterby the ability to perform a given action indicating awareness of theuser's position, regardless of whether the given action is actuallyperformed or not.

The level of awareness of a user's position demonstrated by a virtualcharacter may change over time. This change in the awareness level maybe caused by specific actions of the user, or may occur to varyingdegrees of dependence or independence from actions of the user. In someembodiments, the changes in awareness level over time are discreet. Inother embodiments, the changes in awareness level over time arecontinuous. In still other embodiments, the changes in awareness levelover time may exhibit a combination of discreet and continuous changes.

With reference to FIG. 12A, a graph of awareness versus time is shown,illustrating discreet changes in awareness of a virtual character overtime, in accordance with an embodiment of the invention. As shown, theregions 120 of the graph illustrate a base level of awareness. This baselevel of awareness may in fact indicate that the virtual character willexhibit no awareness of the user's position, or that the virtualcharacter exhibits a minimal amount of awareness of the user's position.At region 121, the virtual character exhibits a first level of awarenessof the user's position. At regions 122, the virtual character exhibits asecond level of awareness of the user's position. And at region 123, thevirtual character exhibits a third level of awareness of the user'sposition. In one embodiment, first, second and third levels of awarenessdefine increasing levels of awareness of the user's position exhibitedby the virtual character. For example, a first level might be defined bythe virtual character looking in the direction of the user. A secondlevel might additionally include the virtual character moving in thedirection of the user. And a third level might further include thevirtual character gesturing in the direction of the user. In variousembodiments of the invention, any particular level of awareness may bedefined by an action or combination of actions, or ability to take suchactions, which demonstrate awareness of the user's position. The variousactions and/or abilities defined by the levels of awareness may beadditive or cumulative for increasing levels in some embodiments, or maybe different for different levels in other embodiments.

With reference to FIG. 12B, a graph of awareness versus time is shown,illustrating changes in the level of awareness of a user exhibited by avirtual character, in accordance with an embodiment of the invention. Asshown, the level of awareness demonstrated by the virtual characterchanges in a continuous manner, with various regions of time in whichthe level of awareness changes or maintains a steady state. For example,at regions 130 and 134 the level of awareness of the virtual characteris increasing. At regions 131 and 135, the level of awareness of thevirtual character peaks, and at regions 132 and 136, the level ofawareness of the virtual character is decreasing. At regions 133 and137, the level of awareness of the virtual character maintains aconstant state.

The foregoing illustrated embodiments are provided by way of exampleonly, and not by way of limitation. In various embodiments of theinvention, the level of awareness exhibited by the virtual character mayvary between any number of levels. The level of awareness may becontinuously changing, exhibit time periods of constant levels ofawareness, exhibit a mixture of continuous and discreet changes, etc. Insome embodiments, the virtual character may exhibit periodic changes inthe level of awareness of a user which is exhibited. For example, thevirtual character may be configured to exhibit oscillatory changesbetween different levels of awareness. Or in other embodiment, thevirtual character may exhibit other types of changes that occur in aperiodic fashion.

Furthermore, the level of awareness or change in such level may be tiedto various aspects of the user experience. For example, if the user isplaying a game, then the level of awareness exhibited by the virtualcharacter may change depending on the stage of the gameplay, or inresponse to an action, selection, or some other activity of the user.Thus by way of example, in one embodiment, during one phase or stage ofa game, the virtual character exhibits a certain level of awareness ofthe user, while during another phase or stage of the game, the virtualcharacter exhibits a different level of awareness. Examples of changesin a phase or stage of gameplay might include such transitions asmovement of the virtual character into a particular area, completion ofan objective, advancement to a new level of the game, beginning a newtask, equipping the virtual character with a particulardevice/item/weapon/etc., or any other kind of change in the gameplaythat may be configured to cause a change in the level of awareness ofthe user exhibited by the virtual character. The transition from onelevel of awareness to another may be discreet or continuous.

In some embodiments, the level of awareness may be determined based onan activity in which the virtual character is engaged. Thus, in oneactivity, the virtual character may exhibit a certain level ofawareness, while for another activity, the virtual character may exhibita different level of awareness. In one embodiment, an awareness profilemay be provided for a given activity, location, stage or otherdeterminant of awareness, the awareness profile indicating levels ofawareness and changes in the level of awareness associated with variousinputs or changes in parameters associated with the virtual character.For example, in one embodiment, when the virtual character is located ina given area, then a corresponding awareness profile might specify thatthe level of awareness of the user exhibited by the virtual charactermay have a certain range. Whereas, when the virtual character is locatedin a different area, the corresponding awareness profile for that areamight specify a different range from the level of awareness exhibited bythe virtual character. This is merely one example of an implementationof an awareness profile, and it will be understood by those skilled inthe art that in other embodiments of the invention, an awareness profilemay be applied to any of various kinds of parameters associated with thevirtual character or the user, such as location, activity, level, stage,input, etc.

With reference to FIG. 12C, a graph illustrating awareness versus timeis shown, in accordance with an embodiment of the invention. A curve138.1 illustrates the level of awareness of a user exhibited by avirtual character over time. In various embodiments of the invention,the awareness of a virtual character as it is monitored over time can beutilized to effect changes in other aspects of a user's experience withan augmented reality environment. For example, an audio volume levelmight be related to the level of awareness. In one embodiment, the audiovolume may be configured to increase as the level of awarenessincreases. Or the audio volume level of a specific portion of theaugmented reality, such as the region containing the virtual character,might be configured to be related to the level of awareness. In anotherembodiment, the zoom level may be related to the level of awareness. Forexample, in one embodiment, when the level of awareness increases, thenthe user's view of the augmented reality scene may zoom in on thevirtual character.

In still other embodiments, various other aspects of the user'sexperience of the augmented reality scene might be affected by thechanges in the level of awareness exhibited by the virtual character.For example, in some embodiments, a color scheme of the augmentedreality scene may be affected by the level of awareness. In oneembodiment, the level of color saturation of the augmented reality sceneor of the virtual character is related to the level of awarenessexhibited by the virtual character. For example, color saturation mayincrease as level of awareness increases. Or in another embodiment,color saturation of regions surrounding the virtual character might beconfigured to decrease as the level of awareness increases, thusaccentuating the visual display of the virtual character within theaugmented reality scene. The foregoing example of aspects of the user'sexperience of the augmented reality scene which can be affected by thelevel of awareness are provided by way of example only, and not by wayof limitation. In other embodiments of the invention, any aspect of theuser's experience of the augmented reality scene may be configured tochange based on changes in the level of awareness.

In various embodiments, the relationship between an aspect of theaugmented reality scene which is affected by the level of awareness, andthe level of awareness may have any kind of relationship. With continuedreference to FIG. 12C, curves 138.2, 138.3, 138.4, and 138.5 illustratecurves for controlling an aspect of the augmented reality scene based onthe level of awareness. For example, an aspect of the augmented realityscene may be directly or linearly correlated to the level of awareness.Curve 138.2 illustrates such an embodiment, the curve 138.2 beinggenerated by adding a constant to the level of awareness curve 138.1.Curve 138.3 also illustrates such a relationship, the curve 138.3 beinggenerated by multiplying the level of awareness curve 138.1 by aconstant. In other embodiments, the relationship of the aspect of theaugmented reality to be controlled is inverse to the level of awareness.Curve 138.4 illustrates such an embodiment, the curve 138.4 representingan inverse of curve 138.1.

In other embodiments of the invention, other types of relationships maybe utilized to control an aspect of the augmented reality experiencebased on the level of awareness exhibited by a virtual character. Forexample, an aspect of the augmented reality experience might becontrolled according to a first-order derivative of the level ofawareness curve. Curve 138.5 illustrates a first-order derivative of thelevel of awareness curve 138.1. In one embodiment, the first-orderderivative is directly utilized to control an aspect of the augmentedreality experience. In another embodiment, the first-order derivativemay be used in a predictive manner, to anticipate a future level ofawareness, and adjust an aspect of the augmented reality experienceaccordingly. The foregoing examples of types of relationships between anaspect of the augmented reality scene and the level of awareness areprovided by way of example only, and not by way of limitation. In otherembodiments of the invention, the relationship may be any type ofrelationship according to which an aspect of the augmented reality sceneis controlled based on the level of awareness.

With reference to FIG. 13A, a method for causing a virtual character toconvey an awareness of a user is shown, in accordance with an embodimentof the invention. At method operation 140, user interactivity with acomputer program is initiated. The computer program is configured toreceive a captured image stream of a real environment taken by a cameraof a portable device that is held or otherwise attached or associatedwith the user, and generate an augmented reality image stream, whereinthe augmented reality image stream includes a virtual character. Theaugmented reality image stream is displayed to the user on the portabledevice. At method operation 142, a determination is made as to whetheror not the user is interacting with the virtual character. If it isdetermined that there is no interaction with the virtual character, thenthe method either halts or continues to check for such interaction untilit is determined that the user is interacting with the virtualcharacter.

If it is determined that the user is interacting with the virtualcharacter, then the method proceeds to method operation 144, in whichsensor data from the user is detected during the user's interaction withthe virtual character. The sensor data may include any of various kindsof data which are useful for determining the position, orientation, andchanges thereof, of the user and/or the portable device. For example,the sensor data may include a captured image stream taken by the cameraof the portable device, data from motion sensing hardware such as anaccelerometer, magnetometer, or gyroscope, and data from other types ofsensors which are useful for tracking the position of the user. Atmethod operation 146, the detected sensor data is used to track theposition of the user during interaction with the virtual character. Atmethod operation 148, the position of the virtual character is detectedduring the user's interaction with the virtual character. And at methodoperation 150, the relative positioning between the user's position andthe position of the virtual character during the interaction with thecharacter by the user is identified.

At method operation 152, the context of the interactivity is identified.The identified context of the interactivity may include any of variousfeatures and attributes relating to the context in which the userinteracts with the virtual character. For example, the context mayinclude such attributes as the time of day, the location of the virtualcharacter, a setting of the interaction such as a phase or stage ofgameplay, the presence or absence of items both real and virtual, etc.At method operation 154, a signaling attribute is identified for thecontext of interactivity. At method operation 156, the signalingattribute is applied to the virtual character to convey an awareness ofthe user by the virtual character during interactivity.

With reference to FIG. 13B, a flow chart illustrating a method for avirtual character to demonstrate that the virtual character knows wherea human is in his world is shown, in accordance with an embodiment ofthe invention. According to the method, the virtual character throughuse of various sensors determines where the human is relative to itself.As the human physically moves around the virtual character, the virtualcharacter performs various acts that demonstrate that the characterknows where the human is in his world. At method operation 158.1 datainput from various sensors is collected. The various sensors may includean accelerometer, global-positioning system (GPS), gyroscope,magnetometer, camera, and other types of sensors useful for determiningposition and movement. The sensors may be included in a portable deviceheld by the user or otherwise colocational with the human. Or in otherembodiments, the sensors may be positioned colocationally with the humanthrough other mechanisms, such as by attachment to the human's body orclothing.

At method operation 158.2, a process to determine the human positionrelative to a virtual world origin is performed. The virtual world maybe positioned and oriented with reference to markers or anchors in thereal world. A positioning method such as SLAM, or some other type ofpositioning method may be applied so as to enable determination of thehuman position relative to the virtual world origin. At method 158.3 thevirtual character's position relative to the human's position isdetermined.

At method operation 158.4, a signaling attribute of the virtualcharacter is oriented toward the human. The signaling attribute is anaction of the virtual character that is performed so as to demonstratethat the character knows where the human is in his world. For example, abody orientation, a head orientation, an eye orientation, a weaponorientation, or a sound orientation, as well as other types of signalattributes, may be oriented toward the human so as to demonstrate thatthe virtual character knows where the human is located.

With reference to FIG. 14, a diagram illustrating components of aportable device 10 is shown, in accordance with an embodiment of theinvention. The portable device 10 includes a processor 160 for executingprogram instructions. A memory 162 is provided for storage purposes, andmay include both volatile and non-volatile memory. A display 164 isincluded which provides a visual interface that a user may view. Abattery 166 is provided as a power source for the portable device 10. Amotion detection module 168 may include any of various kinds of motionsensitive hardware, such as a magnetometer 170, an accelerometer 172,and a gyroscope 174.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 172 are used toprovide the direction of gravity, which gives an absolute reference for2 angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the controller. In one embodiment, threemagnetometers 170 are used within the controller, ensuring an absolutereference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 172 is used together with magnetometer 170 toobtain the inclination and azimuth of the portable device 10.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes provide information about movement across the respective axis(x, y and z) based on inertial sensing. The gyroscopes help in detectingfast rotations. However, the gyroscopes can drift overtime without theexistence of an absolute reference. This requires resetting thegyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 176 is provided for capturing images and image streams of areal environment. More than one camera may be included in the portabledevice 10, including a camera that is rear-facing (directed away from auser when the user is viewing the display of the portable device), and acamera that is front-facing (directed towards the user when the user isviewing the display of the portable device). Additionally, a depthcamera 178 may be included in the portable device for sensing depthinformation of objects in a real environment.

The portable device 10 includes speakers 180 for providing audio output.Also, a microphone 182 may be included for capturing audio from the realenvironment, including sounds from the ambient environment, speech madeby the user, etc. The portable device 10 includes tactile feedbackmodule 184 for providing tactile feedback to the user. In oneembodiment, the tactile feedback module 184 is capable of causingmovement and/or vibration of the portable device 10 so as to providetactile feedback to the user.

LEDs 186 are provided as visual indicators of statuses of the portabledevice 10. For example, an LED may indicate battery level, power on,etc. A card reader 188 is provided to enable the portable device 10 toread and write information to and from a memory card. A USB interface190 is included as one example of an interface for enabling connectionof peripheral devices, or connection to other devices, such as otherportable devices, computers, etc. In various embodiments of the portabledevice 10, any of various kinds of interfaces may be included to enablegreater connectivity of the portable device 10.

A WiFi module 192 is included for enabling connection to the Internetvia wireless networking technologies. Also, the portable device 10includes a Bluetooth module 194 for enabling wireless connection toother devices. A communications link 196 may also be included forconnection to other devices. In one embodiment, the communications link196 utilized infrared transmission for wireless communication. In otherembodiments, the communications link 196 may utilized any of variouswireless or wired transmission protocols for communication with otherdevices.

Input buttons/sensors 198 are included to provide an input interface forthe user. Any of various kinds of input interfaces may be included, suchas buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 200 may be included in portable device 10 forfacilitating communication with other devices via ultra-sonictechnologies.

The foregoing components of portable device 10 have been described asmerely exemplary components that may be included in portable device 10.In various embodiments of the invention, the portable device 10 may ormay not include some of the various aforementioned components.Embodiments of the portable device 10 may additionally include othercomponents not presently described, but known in the art, for purposesof facilitating aspects of the present invention as herein described.

The foregoing portable device, and its included components, constitutemeans for initiating user interactivity with a computer program, andmeans for determining interaction with a virtual character. The portabledevice constitutes means for detecting sensor data from the user duringinteraction with the virtual character, and means for tracking the userposition using the sensor data during the interaction with the virtualcharacter. The portable device constitutes means for detecting theposition of the virtual character during the user's interaction with thevirtual character, and means for identifying a relative position betweenthe user's position and the position o the virtual character duringinteraction with the virtual character by the user.

The portable device constitutes means for identifying a context of theinteractivity, and means for identifying a signaling attribute for thecontext of interactivity. Additionally, the portable device constitutesmeans for applying the signaling attribute to the virtual character toconvey an awareness of the user by the virtual character during theinteractivity.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

What is claimed is:
 1. A method for causing a virtual character todemonstrate awareness of a user's position or activity, the methodcomprising, detecting a position of a portable device, the portabledevice being substantially colocational with the user, the portabledevice including a camera and a display, the camera capturing a firstimage stream of a real-world scene, wherein the detection of theposition of the portable device comprises identification of an anchorwithin the first image stream, the anchor being tracked during movementof the portable device; generating a second image stream based on thefirst image stream, the second image stream comprising an augmentedreality scene formed by inserting a virtual character into thereal-world scene as captured in the first image stream, the virtualcharacter being positioned in the augmented reality scene in relation tothe identified anchor, the virtual character configured to perform anaction indicating awareness of the position or activity of the user;displaying the second image stream on the display of the portabledevice; wherein the action indicating awareness comprises adjusting aview direction, in the augmented reality scene, of the virtualcharacter, the view direction of the virtual character being definedrelative to the anchor that is tracked during the movement of theportable device; wherein the adjustment of the view direction of thevirtual character is in the direction of the position of the portabledevice and is defined by turning of the virtual character from a currentview direction to an adjusted view direction facing the portable device,and wherein the adjustment of the view direction of the virtualcharacter changes a relationship of the view direction to the anchor,wherein the adjustment of the view direction of the virtual characterincludes adjustment of a head or eyes of the virtual character so as tolook in the direction of the portable device.
 2. The method of claim 1,wherein the insertion of the virtual character into the real-world scenecomprises placement of the virtual character on a surface identifiedwithin the real-world scene.
 3. The method of claim 1, wherein theanchor is selected from the group consisting of one or more augmentedreality tags and one or more features within the real-world scene. 4.The method of claim 1, wherein the performance of an action indicatingawareness occurs automatically, without requiring a command from theuser.
 5. The method of claim 1, further comprising, continuouslyupdating the adjustment of the view direction of the virtual characterin the direction of the position of the portable device, as the portabledevice moves in three-dimensional space.
 6. A method for causing avirtual character to demonstrate awareness of a user's position oractivity, the method comprising, capturing a first image stream of areal-world scene at a camera of a portable device, the portable devicebeing substantially colocational with the user; identifying and trackingone or more markers in the first image stream; tracking the position andorientation of the portable device based on the tracking of the one ormore markers; generating a second image stream based on the first imagestream, the second image stream depicting a virtual reality scenepositioned with reference to the one or more markers in the first imagestream, the virtual reality scene including a virtual characterpositioned with reference to the one or more markers in the first imagestream and configured to perform an action indicating awareness of theposition or activity of the user; displaying the second image stream ona display of the portable device; wherein the action indicatingawareness comprises adjusting a view direction, in the virtual realityscene, of the virtual character, the view direction of the virtualcharacter being defined relative to the one or more markers; wherein theadjustment of the view direction is in the direction of the position ofthe portable device, and wherein the adjustment of the view directionchanges a relationship of the view direction to the one or more markers;and continuously updating the adjustment of the view direction as theportable device moves in three-dimensional space.
 7. The method of claim6, wherein the markers are selected from the group consisting of one ormore augmented reality tags and one or more features within thereal-world scene.
 8. The method of claim 6, wherein the performance ofan action indicating awareness occurs automatically, without requiring acommand from the user.
 9. The method of claim 6, wherein the adjustmentof the view direction of the virtual character includes adjustment of ahead or eyes of the virtual character so as to look in the direction ofthe portable device.
 10. A method for providing user interactivity witha virtual character, the method comprising, initiating userinteractivity with a computer program, determining user interaction witha virtual character of the computer program, detecting sensor data fromthe user during the user's interaction with the virtual character,wherein the sensor data includes an image stream captured by a portabledevice controlled by the user; tracking a position of the user using thesensor data, wherein the tracking of the position of the user includesidentification and tracking of one or more markers within the imagestream to enable determination of a position of the portable device;detecting a position of the virtual character during the user'sinteraction with the virtual character; identifying a relativepositioning between the user's position and the position of the virtualcharacter during the user's interaction with the virtual character;identifying a context of the user's interaction with the virtualcharacter; identifying a signaling attribute for the context of theuser's interaction with the virtual character; applying the signalingattribute to the virtual character to convey an awareness of the user bythe virtual character; wherein the computer program provides a virtualreality environment positioned with reference to the one or moremarkers, the virtual reality environment including the virtual characterpositioned with reference to the one or more markers; wherein the userinteractivity with the computer program includes interactivity with thevirtual reality environment; wherein the signaling attribute is selectedfrom the group consisting of adjustment of the virtual character's viewdirection in the virtual reality environment towards the position of theuser so that the adjustment of the view direction changes a relationshipof the view direction to the one or more markers, movement of thevirtual character in the virtual reality environment towards theposition of the user, gesturing by the virtual character in the virtualreality environment towards the position of the user.
 11. The method ofclaim 10, wherein the computer program provides an augmented realityenvironment, the augmented reality environment including the virtualcharacter; and wherein the user interactivity with the computer programincludes interactivity with the augmented reality environment.
 12. Themethod of claim 10, wherein the context of the user's interaction withthe virtual character is selected from the group consisting of alocation within a real-world environment, a location within a virtualenvironment, a stage within the computer program.
 13. A method forproviding interactivity with a virtual character, comprising: capturing,by a camera of a portable device, video of a real-world scene having oneor more anchors; generating an augmented reality scene based on thecaptured image stream, the augmented reality scene including a virtualcharacter positioned in the augmented reality scene in relation to theone or more anchors; rendering the augmented reality scene on a displayof the portable device; tracking a direction of the camera of theportable device; adjusting a view direction of the virtual character,defined relative to the one or more anchors, in response to the trackeddirection of the camera; wherein when the direction of the camera istowards the virtual character, then the view direction of the virtualcharacter is adjusted towards a location of the portable device, andwherein the adjustment of the view direction of the virtual characterchanges a relationship of the view direction to the one or more anchors.14. The method of claim 13, wherein the adjustment of the view directionof the virtual character towards the location of the portable device isdefined by turning of the virtual character from a current viewdirection to an adjusted view direction facing the portable device. 15.The method of claim 14, wherein when the direction of the camera istowards an object in the augmented reality scene other than the virtualcharacter, then the view direction of the virtual character is adjustedtowards the object in the augmented reality scene.